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Abstract 

Network updates such as policy and routing changes 
occur frequently in Software Defined Networks (SDN). 
Updates should be performed consistently, preventing 
temporary disruptions, and should require as little over¬ 
head as possible. Scalability is increasingly becom¬ 
ing an essential requirement in SDN. In this paper 
we propose to use time-triggered network updates to 
achieve consistent updates. Our proposed solution re¬ 
quires lower overhead than existing update approaches, 
without compromising the consistency during the up¬ 
date. We demonstrate that accurate time enables far 
more scalable consistent updates in SDN than previ¬ 
ously available. In addition, it provides the SDN pro¬ 
grammer with fine-grained control over the tradeoff be¬ 
tween consistency and scalability. 

1. INTRODUCTION 
1.1 Background 

Traditional network management systems are in 
charge of initializing the network, monitoring it, and 
allowing the operator to apply occasional changes when 
needed. Software Defined Networking (SDN), on the 
other hand, requires a central controller to routinely 
perform frequent policy and configuration updates in 
the network. 

The centralized approach used in SDN introduces 
challenges in terms of consistency and scalability. The 
controller must take care to minimize network anoma¬ 
lies during update procedures, such as packet drops or 
misroutes caused by temporary inconsistencies. Up¬ 
dates must also be planned with scalability in mind; 
update procedures must scale with the size of the net¬ 
work, and cannot be too complex. In the face of rapid 
configuration changes, the update mechanism must al¬ 
low a high update rate. 

Two main methods for consistent network updates 
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have been thoroughly studied in the last few years. 

• Ordered updates. This approach uses a sequence 
of configuration commands, whereby the order of exe¬ 
cution guarantees that no anomalies are caused in in¬ 
termediate states of the procedure [11, 18, 45, 24]; at 
each phase the controller waits until all the switches 
have completed their updates, and only then invokes 
the next phase in the sequence. 

• Two-phase updates. In the two-phase approach [42, 
20], configuration version tags are used to guarantee 
consistency; in the first phase the new configuration 
is installed in all the switches in the middle of the net¬ 
work, and in the second phase the ingress switches are 
instructed to start using a version tag that represents 
the new configuration. During the update procedure 
every switch maintains two sets of entries: one for the 
old configuration version, and one for the new version. 
The version tag attached to the packet determines 
whether it is processed according to the old configu¬ 
ration or the new one. After the packets carrying the 
old version tag are drained from the network, garbage 
collection is performed on the switches, removing the 
duplicate entries and leaving only the new configura¬ 
tion. 

In previous work [29] we argued that time is a pow¬ 
erful abstraction for coordinating network updates. We 
defined an extension [33] to the OpenFlow protocol [39] 
that allows time-triggered operations. This extension 
has been approved and integrated into OpenFlow 1.5 [41], 
and into the OpenFlow 1.3.x extension package [40]. 

1.2 Time for Consistent Updates 

In this paper we study the use of accurate time to 
trigger consistent network updates. We define a time- 
based order approach, where each phase in the sequence 
is scheduled to a different execution time, and a time- 
based two-phase approach, where each of the two phases 
is invoked at a different time. 

We show how the order and two-phase approaches 
benefit from time-triggered phases. Contrary to the 
conventional order and two-phase approaches, timed 



updates do not require the controller to wait until a 
phase is completed before invoking the next phase, sig¬ 
nificantly simplifying the controller’s involvement in the 
update process, and reducing the update duration. 

The time-based method significantly reduces the time 
duration required by the switches to maintain duplicate 
policy rules for the same flow. In order to accommodate 
the duplicate policy rules, switch flow tables should have 
a set of spare flow entries [42, 20] that can be used for 
network updates. Timed updates use each spare entry 
for a shorter duration than untimed updates, allowing 
higher scalability. 

Accurate time synchronization has evolved over the 
last decade, as the Precision Time Protocol (PTP) [16] 
has become a common feature in commodity switches, 
allowing sub-microsecond accuracy in practical use 
cases (e.g., [4]). However, even if switches have per¬ 

fectly synchronized clocks, it is not guaranteed that up¬ 
dates are executed at their scheduled times. We argue 
that a carefully designed switch can schedule updates 
with a high degree of accuracy. Moreover, we show that 
even if switches are not optimized for accurate schedul¬ 
ing, then the timed approach outperforms conventional 
update approaches. 

The use of time-triggered updates accentuates a 
tradeoff between update scalability and consis¬ 
tency. At one end of the scale, consistent updates come 
at the cost of a potentially long update duration, and 
expensive memory waste due to rule duplication.^ At 
the other end, a network-wide update can be invoked 
simultaneously, using TimeConf [29], allowing a short 
update time, preventing the need for rule duplication, 
but yielding a brief period of inconsistency. In this pa¬ 
per we show that timed updates can be tuned to any 
intermediate point along this scale. 

1.3 Contributions 

The main contributions of this paper are as follows. 


• We show that accurate time provides the SDN pro¬ 
grammer with a knob for fine-tuning the tradeoff be¬ 
tween consistency and scalability. 

• We present an experimental evaluation on a 50-node 
testbed, demonstrating the significant advantage of 
timed updates over other update methods. 

2. TIME-BASED CONSISTENT UPDATES 

We now describe the concept of time-triggered consis¬ 
tent updates. We assume that switches keep local clocks 
that are synchronized to a central reference clock by 
a synchronization protocol, such as the Precision Time 
Protocol (PTP) [16] or ReversePTP [32, 31], or by an 
accurate time source such as GPS. The controller sends 
network update messages to switches using an SDN pro¬ 
tocol such as OpenFlow [41]. An update message may 
specify when the corresponding update is scheduled to 
be performed. 



2.1 Ordered Updates 

Fig. la illustrates an ordered network update. We 
would like to reconfigure the path of a traffic flow from 
the ‘before’ to the ‘after’ configuration. An ordered up¬ 
date proceeds as described in Fig. 2; the phases in the 
procedure correspond to the numbers in Fig. la. 


• We propose to use time-triggered network updates in 
a way that requires a lower overhead than existing 
update approaches without compromising the consis¬ 
tency during the update. 

• We show that timed consistent updates require a shorter 
duration than existing consistent update methods. 

• We define an inconsistency metric, allowing to quan¬ 
tify how consistent a network update is. 

^As shown in [20], the duration of an update can be traded 
for the update rate. The flow table will typically include a 
limited number of excess entries that can be used for dupli¬ 
cated rules. By reducing the update duration, the excess en¬ 
tries are used for a shorter period of time, allowing a higher 
number of updates per second. 


Untimed Ordered Update 

1 Controller sends the ‘after’ configuration to Si. 

2 Controller sends the ‘after’ configuration to S' 2 . 

3 Controller updates S 3 (garbage collection). 


Figure 2: Ordered update procedure for the scenario 
of Fig. la. 

The ordered update procedure guarantees that if ev¬ 
ery phase is performed after the previous phase was 
completed, then no packets are dropped during the pro¬ 
cess. A time-based order update procedure is de¬ 
scribed in Fig. 3. 

Notably, the ordered approach requires the controller 
to be involved in the entire update procedure, making 
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Timed Ordered Update 

0 Controller sends timed updates to all switches. 

1 S'! enables the ‘after’ configuration at time Ti. 

2 S 2 enables the ‘after’ configuration at time T 2 > Ti. 

3 S 3 performs garbage collection at time T 3 > T 2 . 


Figure 3: Timed Ordered update procedure for the sce¬ 
nario of Fig. la. 


the update process sensitive to the load on the con¬ 
troller, and to the communication delays at the time of 
execution. In contrast, in the time-base protocol, the 
controller is only involved in phase 0, and if Ti is timed 
correctly, the update process is not influenced by these 
issues. 

2.2 Two-phase Updates 

An example of a two-phase update is illustrated in 
Fig. lb; the figure depicts a multicast distribution tree 
through a network of three switches. Multicast packets 
are distributed along the paths of the ‘before’ tree. We 
would like to reconfigure the distribution tree to the 
‘after’ state. 


Untimed Two-phase Update 

1 Controller sends the ‘after’ configuration to Si. 

2 Controller instructs S 2 to start using the ‘after’ 
configuration with the new version tag. 

3 Controller updates Ai (garbage collection). 


Figure 4: Two-phase update procedure for the scenario 
of Fig. lb. 

The two-phase procedure [42, 20] is described in Fig. 4. 
In the first phase, the new configuration is installed in 
Si, instructing it to forward packets that have the new 
version tag according to the ‘after’ configuration. In 
the second phase, S 2 is instructed to forward packets 
according to the ‘after’ configuration using the new ver¬ 
sion tag. The ‘before’ configuration is removed in the 
third phase. As in the ordered approach, the two-phase 
procedure requires every phase to be invoked after it is 
guaranteed that the previous phase was completed. 

In the timed two-phase approach, specified in Fig. 5, 
phases 1,2, and 3 are scheduled in advance by the con¬ 
troller. The switches then execute phases 1,2, and 3 at 
times Ti, T2, and T3, respectively. 

2.3 k-Phase Consistent Updates 

The order approach guarantees consistency if updates 
are performed according to a specific order. More gen¬ 
erally, we can view an ordered update as a sequence of 


Timed Two-phase Update 

0 Controller sends timed updates to all switches. 

1 Ai enables the ‘after’ configuration at time Ti. 

2 S 2 enables the ‘after’ configuration with the 
new version tag at time T 2 > Ti. 

3 Ai performs garbage collection at time T 3 > T 2 . 


Figure 5: Timed two-phase update procedure for the 
scenario of Fig. lb. 


k phases, where in each phase j, a set of Nj switches 
is updated. For each phase j, the updates of phase j 
must be completed before any update of phase j -I- 1 is 
invoked. 

The two-phase approach is a special case, where k = 
2; in the first phase all the switches in the middle of the 
network are updated with the new policy, and in the 
second phase the ingress switches are updated to start 
using the new version tag. 

2.4 The Overhead of Network Updates 

Both the order method and the two-phase method re¬ 
quire duplicate configurations to be present during the 
update procedure. In each of the protocols of Fig. 2- 
5, both the ‘before’ and the ‘after’ configurations are 
stored in the switches’ expensive flow tables from phase 1 
to phase 3. The unnecessary entries are removed only 
after garbage collection is performed in phase 3. 

In the timed protocols of Fig. 3 and 5 the switches 
receive the update messages in advance (phase 0), and 
can temporarily store the new configurations in a non- 
expensive memory. The switches install the new con¬ 
figuration in the expensive flow table memories only at 
the scheduled times, thereby limiting the period of du¬ 
plication to the duration from phase 1 to phase 3. 

The overhead cost of the duplication depends on the 
time elapsed between phase 1 and phase 3. Hence, 
throughout the paper we use the update duration as 
a metric for quantifying the overhead of a consistent 
update that includes a garbage collection phase. 

3. TERMINOUOGY AND NOTATIONS 

3.1 The Network Model 

We reuse some of the terminology and notations of [42]. 
Our system consists of A -|- 1 nodes: a controller c, and 
a set of N switches, S = {S'!,..., S^}- A packet is a se¬ 
quence of bits, denoted by pfc G P/c, where Fk is the set 
of possible packets in the system. Every switch S'i G S 
has a set Pr^ of ports. 

The sources and destinations of the packets are as¬ 
sumed to be external; packets are received from the 
‘outside world’ through a subset of the switches’ ports. 
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referred to as ingress ports. An ingress switch is a switch 
that has at least one ingress port. Every packet pk is for¬ 
warded through a sequence of switches (S'q, •. •, 
where the first switch S'ij is an ingress switch. The 
last switch in the sequence, Si^, forwards the packet 
through one of its ports to the outside world. 

When a packet pk is received by a switch Si through 
port p € Pri, the switch uses a forwarding function 
Fi : PfcxPri —A, where A is the set of possible actions 
a switch can perform, e.g., ‘forward the packet through 
port q’. The packet content and the port through which 
the packet was received determine the action that is 
applied to the packet. 

It is assumed that every switch maintains a local 
clock. As is standard in the literature (e.g., [23]), we 
distinguish between real time, an assumed Newtonian 
time frame that is not directly observable, and local 
clock time, which is the time measured on one of the 
switches’ clocks. We denote values that refer to real 
time by lowercase letters, e.g. t, and values that refer 
to clock time by uppercase, e.g., T. 

We dehne a packet instance to be a tuple {pk, Si,p, t), 
where pk G Pfc is a packet, 5^ € S is the ingress switch 
through which the packet is received, p G Pr^ is the 
ingress port at switch Si, and t is the time at which the 
packet instance is received by Si. 

3.2 Network Updates 

We define a singleton update u of switch Si to be a 
partial function, u : Vk x Pr^ ^ A. A switch applies 
a singleton update, u, by replacing its forwarding func¬ 
tion, Fi with a new forwarding function, F', that be¬ 
haves like u in the domain of u, and like F^ otherwise. 

We assume that every singleton update is triggered by 
a set of one or more messages sent by the controller to 
one of the switches. 

We define an update U to be a set of singleton updates 
U = {Ui, . . .,Um}- 

We define an update procedure, U, to be a set 
U = {{ui,ti,phase{ui)),..., {um,tm,phase{um))} of 3- 
tuples, such that for all 1 < j < m, we have that Uj is a 
singleton update, phase{uj) is a positive integer specify¬ 
ing the phase number of Uj, and tj is the time at which 
Uj is performed. Moreover, it is required that for every 
1 < hj < m if phase{ui) < phase{uj) then A < tj. 
This dehnition implies that an update procedure is a 
sequence of one or more phases, where each phase is 
performed after the previous phase is completed, but 
there is no guarantee about the order of the singleton 
updates of each phase. 

A fc-phase update procedure is an update procedure 
U = {{ui,ti,phase{ui )),..., {um,tm,phase{um))} in which 
for all 1 < j < m we have 1 < phase{uj) < k, and 
for all 1 < z < fc there exists an update Uj such that 
{uj,tj,i) G U. 


We define a timed singleton update to be a pair 
{u,T), where zz is a singleton update, and T is a clock 
value that represents the scheduled time of u. We as¬ 
sume that every switch maintains a local clock, and that 
when a switch receives a message indicating a timed sin¬ 
gleton update u’^ it implements the update as close as 
possible to the instant when its local clock reaches the 
value T. Similar to the definition of an update proce¬ 
dure, we define a timed update proeedure to be a set 
= {{u]{,ti,phase{ui)),..., (zz^, phase (zz^))}. 

An update procedure U = 

{{ui,ti,phase{ui)),..., (zz^, phase (zz™))} 
and a timed update procedure = 

{(z;^i,ti,phase(z;^i)),..., (z;^„, t„,phase(z)^„))} = 

{((z)i,Ti),ti,phase(v^i)),..., ((z;„, T„), f„,phase(z)^„))} 
are said to be similar, denoted by ~ U if to = n 
and for every 1 < j < m we have Uj = Vj and 
phase{uj) = phase{vj). 

Given an untimed update, U, the original conhgura- 
tion, before any of the singleton updates of U takes 
place, is given by the set of forwarding functions, 
{Fi,... ,Fjv}. We denote the new configuration, after 
all the singleton updates of U have been implemented, 
hy {¥[,..., 

We define consistent forwarding based on the per- 
packet consistency definition of [42]. Intuitively, a 
packet is consistently forwarded if it is processed either 
according to the new configuration or according to the 
old one, but not according to a mixture of the two. For¬ 
mally, let {pk, Si^,pi,t) be a packet instance that is for¬ 
warded through a sequence of switches Si^, Si.^,..., Si^ 
through ports pi,p2 ,... ,Pm, respectively, and is as¬ 
signed the actions ai, a2 ,..., am,. The packet instance 
{pk, 5'i J, Pi , t) is said to be consistently forwarded if one 
of the following is satisfied: 

(i) ¥i^{pk,pj) = Gj for all I < j < to, or 

(ii) ¥'i.{pk,pj) = Uj for all I < j < to. 

A packet instance that is not consistently forwarded, 
is said to be ineonsistently forwarded. 

3.3 Delay-related Notations 

Table 1 presents key notations related to delay and 
performance. The attributes that play a key role in 
our analysis are Dc, Dn, and <5. These attributes are 
discussed further in Section 4. 

4. UPPER AND LOWER BOUNDS 
4.1 Delay Upper Bounds 

Both the order and the two-phase approaches implic¬ 
itly assume the existence of two upper bounds, Dc and 
Dn (see Table 1): 

• Dc'. both approaches require previous phases in the 

update procedure to be completed before invoking the 

current phase. Therefore, after sending an update 
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Dc 

An upper bound on the controller-to- 
switch delay^ including the network la¬ 
tency, and the internal switch delay until 
completing the update. 

Dn 

An upper bound on the end-to-end network 
delay. 

A 

An upper bound on the time interval be¬ 
tween the transmission times of two con¬ 
secutive update messages sent by the con¬ 
troller. 

(5 

An upper bound on the scheduling error; 
an update that is scheduled to be per¬ 
formed at T is performed in practice during 
the time interval [T, T -|- ^]. 

Tsu 

The timed update setup time; in order to 
invoke a timed update that is scheduled 
to time T, the controller sends the update 
messages no later than at T — 


Table 1: Delay-related Notations 


message, the controller must wait for a period of Dc 
until it is guaranteed that the corresponding update 
has been performed; only then can it invoke the next 
phase in the procedure. Alternatively, explicit ac¬ 
knowledgments can be used to indicate update com¬ 
pletions; when a switch completes the update it noti¬ 
fies the controller. Unfortunately, OpenFlow [41, 26] 
currently does not support such an acknowledgment 
mechanism. Hence, one can either use other SDN 
protocols that support explicit acknowledgment (as 
was assumed in [18]), or wait for a period of until 
the switch is guaranteed to complete the update. 

• Dn'. garbage collection can take place after the up¬ 
date procedure has completed, and all en-route pack¬ 
ets have been drained from the network. Garbage 
collection can be invoked either after waiting for a 
period of D„ after completing the update, or by us¬ 
ing soft timeoutsf Both of these approaches assume 
there is an upper bound, D„, on the end-to-end net¬ 
work latency. 

Is it practical to assume that the upper bounds 
Dc and exist? Network latency is often modeled 
using long-tailed distributions such as exponential or 
Gamma [37, 13], implying that network latency is often 

unbounded. 

We demonstrate the long-tailed behavior of network 
latency by analyzing measurements performed on pro¬ 
duction networks. We analyze 20 delay measurement 

^Soft timeouts are defined in the OpenFlow protocol [41] as a 
means for garbage collection; a flow entry that is configured 
with a soft timeout, is cleared if it has not been used 
for a duration Dn. 


datasets from [6, 2] taken at various sites over a one- 
year period, from November 2013 to November 2014. ^ 
The measurements capture the round-trip time (RTT) 
using ICMP Echo requests. The measurements show 
(Fig. 6) that in some networks the 99.999*^ percentile 
is almost two orders of magnitude higher than the av¬ 
erage RTT. Table 2 summarizes the ratio between tail 
latency values and average values in the 20 traces we 
analyzed. 
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Figure 6: Long-tail latency 


99.9*'* 

percentile 

99.99*'* 

percentile 

99.999*'* 

percentile 

4.88 

10.49 

19.45 


Table 2: The mean ratio between the tail latency and 
the average latency. 

In typical networks we expect Dn to have long-tailed 
behavior. Similar long-tailed behavior has also been 
shown for Dc in [18, 43]. 

At a first glance, these results seem troubling: if net¬ 
work latency is indeed unbounded, neither the order nor 
the two-phase approaches can guarantee consistency, 
since the controller can never be sure that the previous 
phase was completed before invoking the next phase. 

In practice, typical approaches will not require a true 
upper bound, but rather a latency value that is exceeded 
with a sufficiently low probability. Service Level Agree¬ 
ment (SLA) in carrier networks is a good example of 
this approach; per the MEF 10.3 specification [27], a 
Service Level Specification (SLS) defines not only the 
mean delay, but also the Frame Delay Range (FDR), 
and the percentile defining this range. Thus, service 
providers must guarantee that the rate of frames that 
exceed the delay range is limited to a known percentage. 

^Details about the measurements can be found in Ap¬ 
pendix A. 
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Throughout the paper we use Dc and Z)„, referring to 
the upper bounds of the delays. In practice, these may 
refer to a sufficiently high percentile delay. Our analysis 
in Section 6 revisits the upper bound assumption. 

4.2 Delay Lower Bounds 

Throughout the paper we assume that the lower 
bounds of the network delay and the controller-to- 
switch delay are zero. This assumption simplifies the 
presentation, although the model can be extended to 
include non-zero lower bounds on delays. 

4.3 Scheduling Accuracy Bound 

As defined in Table 1, 6 is an upper bound on the 
scheduling error, indicating how accurately updates are 
scheduled; an update that is scheduled to take place 
at time T is performed in practice during the interval 
[r ,T + 5]^ A switch’s scheduling accuracy depends on 
two factors: (i) how accurately its clock is synchronized 
to the system’s reference clock, and (ii) its ability to 
perform real-time operations. 

Most high-performance switches are implemented as 
a combination of hardware and software components. A 
scheduling mechanism that relies on the switch’s soft¬ 
ware may be affected by the switch’s operating system 
and by other running tasks, consequently affecting the 
scheduling accuracy. Furthermore, previous work [18, 
43] has shown high variability in rule installation laten¬ 
cies in Ternary Content Addressable Memories (TC AMs), 
resulting from the fact that a TCAM update might re¬ 
quire the TCAM to be rearranged. 

Nevertheless, existing switches and routers practice 
real-time behavior, with a predictable guaranteed re¬ 
sponse time to important external events. Traditional 
protection switching and fast reroute mechanisms re¬ 
quire the network to react to a path failure in less than 
50 milliseconds, implying that each individual switch or 
router reacts within a few milliseconds, or in some cases 
less than one millisecond (e.g. [38]). Operations, Ad¬ 
ministration, and Maintenance (0AM) protocols such 
as the IEEE 802.lag [1] require faults to be detected 
within a strict timing constraint of ±0.42 milliseconds.^ 
Measures can be taken to implement accurate 
scheduling of timed updates: 

• Common real-time programming practices can be ap¬ 
plied to ensure guaranteed performance for time-based 
update, by assigning a constant fraction of time to 
timed updates. 

• When a switch is aware of an update that is sched¬ 
uled to take place at time T^, it can avoid perform- 

^An alternative representation of 5 assumes a symmetric 
error, T ±5/2. The two approaches are equivalent. 

® Faults are detected using Continuity Check Messages 
(CCM), transmitted every 3.33 ms. A fault is detected when 
no CCMs are received for a period of 11.25 ± 0.42 ms. 


ing heavy maintenance tasks near this time, such as 
TCAM entry rearrangement. 

• Untimed update messages received slightly before time 
Ts can be queued and processed after the scheduled 
update is executed. 

• If a switch receives a time-based command that is 
scheduled to take place at the same time as a pre¬ 
viously received command, it can send an error mes¬ 
sage to the controller, indicating that the last received 
command cannot be executed. 

• It has been shown that timed updates can be sched¬ 
uled with a very high degree of accuracy, on the or¬ 
der of 1 microsecond, using TimeFlip [34]. This ap¬ 
proach provides a high scheduling accuracy, poten¬ 
tially at the cost of some overhead in the switch’s 
flow tables. 

Observation 1. In typical settings S < Dc- 

The intuition behind Observation 1 is that 5 is only 
affected by the switch’s performance, whereas Dc is af¬ 
fected by both the switch’s performance and the net¬ 
work latency. We expect Observation 1 to hold even 
if switches are not designed for real-time performance. 
We argue that in switches that use some of the real-time 
techniques above, S << Dc, making the timed approach 
significantly more advantageous, as we shall see in the 
next section. 

5. WORST-CASE ANALYSIS 

5.1 Worst-case Update Duration 

We define the duration of an update procedure to 
be the time elapsed from the instant at which the first 
switch updates its forwarding function to the instant at 
which the last switch completes its update. 

We use Program Evaluation and Review Technique 
(PERT) graphs [25] to illustrate the worst-case update 
duration analysis. Fig. 7 illustrates a PERT graph of an 
untimed ordered /c-phase update, where three switches 
are updated in each phase. Switches Si, S 2 , and S 3 are 
updated in the first phase, S 4 , S 5 , and Sq are updated 
in the second phase, and so on. In this procedure, the 
controller waits until phase j is guaranteed to have been 
completed before starting phase j ± 1. 

Each node in the PERT graph represents an event, 
and each edge represents an activity. A node labeled 
Cj^i represents the event ‘the controller starts transmit¬ 
ting a phase j update message to switch Si\ A node la¬ 
beled Sj^i represents ‘switch Si has completed its phase 
j update’. The weight of each edge indicates the maxi¬ 
mal delay to complete the transition from one event to 
another. Cstart and Cfin represent the start and fin¬ 
ish times of the update procedure, respectively. The 
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Figure 7: PERT graph of a fc-phase update. 


worst-case duration between two events is given by the 
longest path between the two corresponding nodes in 
the graph. 

Throughout the section we focus on greedy update 
procedures. An update procedure is said to be greedy if 
the controller invokes each update message at the earli¬ 
est possible time that guarantees that for every phase j 
all the singleton updates of phase j are completed before 
those of phase j -I- 1 are initiated. 

5.2 Worst-case Analysis of Untimed Updates 

5.2.1 Untimed Updates 

We start by discussing untimed fc-phase update pro¬ 
cedures, focusing on a single phase, j, in which Nj 
switches are updated. In Lemma 1 and in the upcoming 
lemmas in this section we focus on greedy updates. 

Lemma 1. // U is a multi-phase update proeedure, 
then the worst-case duration of phase j ofU is: 

(iV, - 1). A + i?, (1) 

Proof. Assume that the controller transmits the first 
update message of phase j at time t. Since there is 
no lower bound on the controller-to-switch delay, the 
earliest possible time at which the first switch com¬ 
pletes its update is t. Since Nj switches take part in 
phase j, and A is the upper bound on the duration be¬ 
tween two consecutive messages, the controller invokes 
the last update message of phase j no later than at 
t -\- {Nj — 1) • A. Since Dc is the upper bound on the 
controller-to-switch delay, the update is completed at 
most Dc time units later. Hence, the worst-case update 
duration is {Nj — 1) • A -|- Dc- □ 

The following lemma specifies the worst-case update 
duration of a fc-phase update. The intuition is straight¬ 
forward from Fig. 7. 

Lemma 2. The worst-case update duration of a 
k-phase update procedure is: 

k 

'^{Nj — 1) • A -I- (fc — 1) • max(A,Dc) -b Dc (2) 

j=i 


Proof. Each phase j delays the controller for 
{Nj — 1) • A. Since the update is greedy, at the end 
of each of the first k — 1 phases the controller waits 
max(A, Dc) time units to guarantee that the phase has 
completed, and then immediately proceeds to the next 
phase. The update is completed, in the worst case, Dc 
time units after the controller sends the last update mes¬ 
sage of the phase. The claim follows. □ 

Specifically, in two-phase updates k = 2, yielding: 

Corollary 1. //U is a two-phase update procedure, 
then its worst-case update duration is: 

{Ni + N 2 - 2 )-A + max(A, Dc) + Dc (3) 

5.2.2 Untimed Updates with Garbage Collection 

In some cases, garbage collection is required for some 
of the phases in the update procedure. For example, in 
the two-phase approach, after phase 2 is completed and 
all en-route packets have been drained from the net¬ 
work, garbage collection is required for the Ni switches 
of the first phase. 

More generally, assume that at the end of every phase j 
the controller performs garbage collection for a set of 
Noj switches. Thus, after phase j is completed the 
controller waits Dn time units for the en-route packets 
to drain, and then invokes the garbage collection pro¬ 
cedure for the Ncj switches. 

After invoking the last message of phase j, the con¬ 
troller waits for max(A, Dc -b Dn) time units. Thus, 
the worst-case duration from the transmission of the 
last message of phase j until the garbage collection of 
phase j is completed is given by Eq. 4. 

max(A, Dc -b £>„) -b {Noj - 1) • A -b L>c (4) 

Fig. 8 depicts a PERT graph of a two-phase update 
procedure that includes a garbage collection phase. At 
the end of the second phase, garbage collection is per¬ 
formed for the phase 1 policy rules of ^i, S' 2 , and S 3 . 
This is in fact a special case of a 3-phase update proce¬ 
dure, where the third phase takes place only after all the 
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en-route packets are guaranteed to have been drained 
from the network. The main difference between this ex¬ 
ample and the general fc-phase graph of Fig. 7 is that 
in Fig. 8 the controller waits at least max{A, Dc + Dn) 
time units from the transmission of the last message of 
phase 2 until starting to invoke the garbage collection 
phase. 

Lemma 3. //U is a two-phase update procedure with 
a garbage collection phase, then its worst-case update 
duration is: 

{Ni -|- 7 V 2 -I- Nqi — 3) • a -|- max(A, Dc)+ 

-I- max(A, Dc Dn) Dc 

Proof. In each of the three phases the con¬ 
troller waits at most A time units between two 
consecutive update messages, summing up to 
{Ni -f A 2 -I- Nqi — 3) • a. The controller waits for 
max(A,Hc) time units at the end of phase 1, guar¬ 
anteeing that all the updates of phase 1 have been 
completed before invoking phase 2. At the end of 
phase 2 the controller waits for max(A, Dc Dn) time 
units, guaranteeing that phase 2 is completed, and 
that all the en-routed packets have been drained before 
starting the garbage collection phase. Finally, Dc time 
units after the controller sends the last message of the 
garbage collection phase, the last update is guaranteed 
to be completed. □ 

5.3 Worst-case Analysis of Timed Updates 

5.3.1 Worst-case-based Scheduling 

Based on a worst-case analysis, an SDN pro¬ 
gram can determine an update schedule, T = 
(Ti,..., Tfc, ..., Tg^). Every timed update u* is 
performed no later than at t S. Consequently, we 
can derive the worst-case scheduling constraints below. 

Definition 1 (Worst-case scheduling). If U 
is a timed k-phase update procedure, a schedule T = 
(Ti,..., Tfc, ..., Tg^) is said to he a worst-case 
schedule if it satisfies the following two equations: 

Tj = Tj_i -\- 6 for every phase 2 < j < k (6) 


'^gj - Tj -\- 6 -{- Dn (7) 

for every phase j that requires garbage collection 

Note that a greedy timed update procedure uses worst- 
case scheduling. 

Every schedule T that satisfies Eq. 6 and 7 guaran¬ 
tees consistency. For example, the timed two-phase up¬ 
date procedure of Fig. 9 satisfies the two scheduling 
constraints above. 

5.3.2 Timed Updates 

A timed update starts with the controller sending 
scheduled update messages to all the switches, requiring 
a setup time Tgu- Every phase is guaranteed to take no 
longer than <5. An example of a timed two-phase update 
is illustrated in Fig. 9. 

Lemma 4. The worst-case update duration of a k- 
phase timed update procedure with a worst-case schedule 
is k • 6 . 

Proof. The lemma follows directly from the worst- 
case scheduling constraints of Eq. 6 and 7. □ 

Based on the latter, we derive the following lemma. 

Lemma 5. // U is a two-phase timed update proce¬ 
dure with a garbage collection phase using a worst-case 
schedule, then its worst-case update duration is Dn -I- 3 • 5. 

Proof. By Lemma 4, the first two phases take 2 • 6 
time units. The garbage collection phase requires S ad¬ 
ditional time units, and also time units to allow all 
en-route packets to drain from the network. Thus, the 
update duration is -f 3 • d. □ 

5.4 Timed vs. Untimed Updates 

We now study the conditions under which the timed 
approach outperforms the untimed approach. 

Based on Lemmas 2 and 4, we observe that a timed k- 
phase update procedure has a shorter update duration 
than a similar untimed /c-phase update procedure if: 
















k 

k - 6 < '^{Nj — 1) • A + (fc — 1) • max(A, Dc) + Dc (8) 
i=i 

Lemma 6. Let be a greedy timed k-phase update 
procedure, with a worst-case update duration Di. Let 
V be a greedy untimed k-phase update procedure with a 
worst-case update duration I?2- If S < and ~ U, 
then Di < 02 - 

Proof. By Lemma 4, we have Di = k ■ S. Lemma 2 

k 

yields D 2 = Yf — 1) • A + (fc — 1) • max(A, D^) + D^- 
i=i 

Thus, Di = k ■ 5 < k ■ Dc < {k — 1) ■ max(A, Dc) + Dc 

k 

< E - 1) • A + (A: - 1) • max(A, Dc) + Dc = £>2- It 
j=i 

follows that Di < D 2 . □ 

Now, based on Lemma 3 and Lemma 5, we observe 
that a timed two-phase update procedure with garbage 
collection has a shorter update duration than a similar 
untimed two-phase update procedure if: 

£„ + 3-(5< (Ai+iV2 + AGi-3)-A+ 

+ max(A, Dc) + max(A, Dc + Dn) + Dc 

Lemma 7. Let be a greedy timed two-phase up¬ 
date procedure with a garbage collection phase, with a 
worst-case update duration Di. Let V be a greedy un¬ 
timed two-phase update procedure with a worst-case up¬ 
date duration £>2- If d < Dc and ~ U, then Di < 

D 2 - 

Proof. By Lemma 5 we have Di = Dn -|- 3 • 5, and 
by Lemma 3 we have £>2 = (IVi -\- N 2 Nqi — 3) • A -T 
max(A, Dc) -\- max(A, Dc -f £„) -f Dc- 

Thus, D\ = Dn -|- 3 • 5 < Dn -b 3 • Dc < (fVi -b N 2 -b 
Agi- 3)-A-b£i„-b3-£'c < (fVi-bA2-bfVGi-3)-A-b 
max(A, Dc) -b max(A, Dc -b Dn) -b £c = £*2- It follows 
that Di < £>2, as claimed. □ 

We have shown that it 6 < Dc the timed approach 
yields a shorter update duration than the untimed ap¬ 
proach, and is thus more scalable. Based on Observa¬ 
tion 1, even if switches are not designed for real-time 


performance we have 6 < Dc- We conclude that the 

timed approach is the superior one in typical 
settings. 

6. TIME AS A CONSISTENCY KNOB 

6.1 An Inconsistency Metric 

As discussed in Section 4, the upper bounds Dc and 
Dn do not necessarily exist, or may be very high. Thus, 
in practice consistent network updates only guarantee 
consistent forwarding with a high probability, raising 
the need for a way to measure and quantify to what 
extent an update is consistent. 

Definition 2 (Test flow). A set of packet in¬ 
stances PI is said to he a test flow if for every two packet 
instances {pki, Si,pi,ti) G PI and (pk 2 , S 2 ,P 2 ,t 2 ) G PI, 
all the following conditions are satisfied: 

• Si = S2. 

• Pl= P2- 

• pki = pfc2-® 

• Packet instances are received at a constant packet 
arrival rate R, i.e., if both t 2 > ti and there is 
no packet instance (pk^, S^jp^jt^) G PI such that 
t 2 > h > ti, then t 2 = ti -\-1/R. 

We assume a method that, for a given test flow / and 
an update u, allows to measure the number of packets 
n{f,u) that are forwarded inconsistently.^ 

Definition 3 (Inconsistency metric). Let f he 
a test flow with a packet arrival rate R{f)- Let U be 
an update, and let n{f, U) be the number of packet in¬ 
stances of f that are forwarded inconsistently due to 

®For simplicity, we define that all packets of a test flow are 
identical. It is in fact sufficient to require that all packets 
of the flow are indistinguishable by the switch forwarding 
functions, for example, that all packets of a flow have the 
same source and destination addresses. 

^This measurement can be performed, for example, by per- 
flow match counters in the switches. 
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update U. The inconsistency I{f,U) of a flow f with 
respect to U is defined to be: 

( 10 ) 

The inconsistency I{f,U) is measured in time units. 
Intuitively, I{f,U) quantifies the amount of time that 
flow / is disrupted by the update. 

6.2 Fine Tuning Consistency 

Timed updates provide a powerful mechanism 
that allows SDN programmers to tune the de¬ 
gree of consistency. By setting the update times 
Ti,T 2 , ... ,Tk,Tg-^,... ,Tgi^, the controller can play with 
the consistency-scalability tradeoff; the update over¬ 
head can be reduced at the expense of some inconsis¬ 
tency, or vice versa.® 

Example 1. We consider a two-phase update with 
a garbage collection phase. We assume that (5 = 0 and 
that all packet instances are subject to a constant net¬ 
work delay, Dn- By assigning T = Ti = T 2 = Tg.^, 
the controller schedules a simultaneous update. This 
approach is referred to as TimeConf in [29]. All 
switches are scheduled to perform the update at the same 
time T. Packets entering the network during the period 
[T — Dn,T] are forwarded inconsistently. The incon¬ 
sistency metric in this example is I = Dn. The ad¬ 
vantage of this approach is that it completely relieves 
the switches from the overhead of maintaining duplicate 
entries between the phases of the update procedure. 

Example 2. Again, we consider a two-phase update 
(Fig. 10), with i5 = 0 and a constant network delay, Dn- 
We assign T 2 = Ti -\- d according to Eq. 6, and Tg.^ is 
assigned to be T 2 -\- 6 -\- d, where d < Dn • The update is 
illustrated in the PERT graph of Eig. 10. Hence, packets 
entering the network during the period [T 2 — Dn -f d, T2] 
are forwarded inconsistently. The inconsistency metric 
is equal to I = min{Dn — d,0). In a precise sense, the 
delay d is a knob for tuning the update inconsistency. 

7. EVALUATION 

Our evaluation was performed on a 50-node testbed in 
the DeterLab [44, 28] environment. The nodes (servers) 
in the DeterLab testbed are interconnected by a user- 
configurable topology. 

Each testbed node in our experiments ran a software- 
based OpenFlow switch that supports time-based up¬ 
dates, also known as Scheduled Bundles [41]. A separate 

®In some scenarios, such as security policy updates, even a 
small level of inconsistency cannot be tolerated. In other 
cases, such as path updates, a brief period of inconsistency 
comes at the cost of some packets being dropped, which can 
be a small price to pay for reducing the update duration. 



Figure 10: Example 2: PERT graph of a timed 
two-phase update. The delay d (red in the figure) is a 
knob for consistency. 

machine was used as a controller, which was connected 
to the switches using an out-of-band control network. 

The OpenFlow switches and controller we used are 
a version of OFSoftSwitch and Dpctl [3], respectively, 
that supports Scheduled Bundles [33]. We used Re- 
versePTP [31, 32] to guarantee synchronized timing. 

7.1 Experiment 1: Timed vs. Untimed 
Updates 

We emulated a typical leaf-spine topology (e.g., [9]) 
of N switches, with ^ leaf switches, and y spine 
switches. The experiments were run using various val¬ 
ues of N, between 6 and 48 switches. 



We measured the delay upper bounds, Dn, D^, S, and 
A. Table 3 presents the 99.9*^ percentile delay values 
of each of these parameters. These are the parameters 
that were used in the controller’s greedy updates. 


Dn 

Dc 

(5 

A 

0.262 

4.865 

1.297 

5.24 


Table 3: The measured 99.9*^ percentile of each of the 
delay attributes in milliseconds. 

We observed a low network delay Dn, as it was mea¬ 
sured over two hops of a local area network. In Experi¬ 
ment 2 we analyze networks with a high network delay. 
Note that the values of 6 and Dc were measured over 
software-based switches. Since hardware switches may 
yield different values, some of our experiments were per¬ 
formed with various synthesized values of 6 and D^,, as 
discussed below. The measured value of A was high, on 
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Figure 11: Publicly available network topologies [7] used in our experiments. Each node in the graph represents an 
OpenFlow switch. 





Figure 12: Test flows: each path of the test flows in our experiment is depicted by a different color. Black nodes are 
OpenFlow switches. White nodes represent the external source and destination of the test flows in the experiment. 


the order of 5 milliseconds, as Dpctl is not optimized 
for performance. 

The experiments consisted of 3-phase updates of a 
policy rule: (i) a phase 1 update, involving all the 
switches, (ii) a phase 2 update, involving only the leaf 
(ingress) switches, and (iii) a garbage collection phase, 
involving all the switches. 

Results. Fig. 14a compares the update duration of 
the timed and untimed approaches as a function of N. 
Untimed updates yield a significantly higher update du¬ 
ration, since they are affected by (A^i -|-1V2 -I-A^gi —3) • A, 
per Lemma 3.® Hence, the advantage of the timed 

®The slope of the untimed curve in Fig. 14a is A, by 




(a) The update duration as a 
function of the number of 
switches. 


(b) The update duration as a 
function of Dc — <5, for N = 12, 
(5 = 100 ms, various values of Dc 


approach increases with the number of switches 

in the network, illustrating its scalability. 

Fig. 14b shows the update duration of the two ap¬ 
proaches as a function of — 5, as we ran the exper¬ 
iment with synthesized values of 5 and Dc- We fixed 
i5 at 100 milliseconds, and tested various values of Dc- 
As expected (by Section 5.4), the results show that for 
Dc — S > 0 the timed approach yields a lower update 
duration. Furthermore, only when the scheduling error, 
<5, is significantly higher than Dc does the untimed ap¬ 
proach yield a shorter update duration. As discussed in 
Section 4.3, we typically expect Dc — S to be positive, 
as 6 is unaffected by high network delays, and thus we 
expect the timed approach to prevail. Interestingly, the 
results show that even when the scheduling is not 
accurate, e.g., if 6 is 100 milliseconds worse than Dc, 
the timed approach has a lower update duration. 

7.2 Experiment 2: Fine Tuning Consistency 

The goal of this experiment was to study how time 
can be used to tune the level of inconsistency during 
updates. In order to experiment with real-life wide area 
network delay values, D„, we performed the experiment 
using publicly available topologies. 

Network topology. Our experiments ran over three 
publicly available service provider network topologies [7], 

Lemma 3. The theoretical curve was computed based on 
the 99.9*^ percentile value, whereas the mean value in our 
experiment was about 20% lower, explaining the different 
slopes of the theoretical and experimental curves. 


Figure 14: Timed updates vs. untimed updates. Each 
figure shows the experimental values, and the theoreti¬ 
cal worst-case values, based on Lemmas 3 and 5. 
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(a) Sprint - constant network delay. 



Update Duration [milliseconds] 


(b) NetRail - constant network delay. 



Update Duration [milliseconds] 


(c) CompuServe - constant network delay. 



(d) Sprint - exponential network delay. 


(e) NetRail - exponential network delay, (f) CompuServe - exponential network delay. 


Figure 15: Inconsistency as a function of the update duration. Modifying the update duration controls the degree 
of inconsistency. Two graphs are shown for each of the three topologies: exponential delay, constant delay. 


as illustrated in Fig. 11. We defined each node in the 
figure to be an OpenFlow switch. OpenFlow messages 
were sent to the switches by a controller over an out-of- 
band network (not shown in the figures). 

Network delays. The public information provided 
in [7] does not include the explicit delay of each path, 
but includes the coordinates of each node. Hence we 
derived the network delays from the beeline distance 
between each pair of nodes, assuming 5 microseconds 
per kilometer, as recommended in [17]. The DeterLab 
testbed allows a configurable delay value to be assigned 
to each link. We ran our experiments in two modes: 

(i) Constant delay — each link had a constant de¬ 
lay that was configured to the value we computed as 
described above. 

(ii) Exponential delay — each link had an exponen¬ 
tially distributed delay. The mean delay of each link 
in experiment (ii) was equal to the link delay of this 
link in experiment (i), allowing an ‘apples to apples’ 
comparison. 

Test flows. In each topology we ran five test flows, 
and measured the inconsistency during a timed net¬ 
work update. Each test flow was injected by an ex¬ 
ternal source (see 12) to one of the ingress switches, 
forwarded through the network, and transmitted from 
an egress switch to an external destination. Test flows 
were injected at a fixed rate of 40 Mbps using Iperf [5]. 

Network updates. We performed two-phase up¬ 
dates of a Multiprotocol Label Switching (MPLS) label; 
a flow is forwarded over an MPLS Label-Switched Path 
(LSP) with label A, and then reconfigured to use label 
B. A garbage collection phase was used to remove the 


entries of label A. Conveniently, the MPLS label was 
also used as the version tag in the two-phase updates. 

Inconsistency measurement. Eor every test flow /, 
and update U, we measure the number of inconsistent 
packets during the update n{f,U). Inconsistent pack¬ 
ets in our context are either packets with a ‘new’ label 
arriving to a switch without the ‘new’ rule, or pack¬ 
ets with an ‘old’ label arriving to a switch without the 
‘old’ configuration. We used the switches’ OpenFlow 
counters to count the number of inconsistent packets, 
n(/, U). We compute the inconsistency of each update 
using Eq. 10. 

Results. We measured the inconsistency I during 
each update as a function of the update duration, Tg^ — 
Ti. We repeated the experiment for each of the topolo¬ 
gies and each of the test flows of Fig. 12. 

The results are illustrated in Fig. 15. The figure de¬ 
picts the tradeoff between the update duration, and the 
inconsistency during the update. A long update dura¬ 
tion bares a cost on the switches’ expensive memory re¬ 
sources, whereas a high degree of inconsistency implies 
a large number of dropped or misrouted packets. 

Using a timed update, it is possible to tune the dif¬ 
ference Tg^ — Ti, directly affecting the degree of incon¬ 
sistency. An SDN programmer can tune Tg^ — Ti to 
the desired sweet spot based on the system constraints; 
if switch memory resources are scarce, one may reduce 
the update duration and allow some inconsistency. 

As illustrated in Fig. 15d, 15e, and 15f, this fine tun¬ 
ing is especially useful when the network latency has 
a long-tailed distribution. A truly consistent update, 
where / = 0, requires a very long and costly update du- 
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ration. As shown in the figures, by slightly compromis¬ 
ing I, the switch memory overhead during the update 
can be cut in half. 

8. DISCUSSION 

Failures. Switch failures during an update proce¬ 
dure may compromise the consistency during an up¬ 
date. For example, a switch may silently fail to per¬ 
form an update, thereby causing inconsistency. Both 
the timed and untimed update approaches may be af¬ 
fected by failure scenarios. The OpenFlow Scheduled 
Bundle [41] mechanism provides an elegant mechanism 
for mitigating failures in timed updates; if the controller 
detects a switch failure before an update is scheduled to 
take place, it can send a cancellation message to all the 
switches that take part in the scheduled update, thus 
guaranteeing an all-or-none behavior. 

Explicit acknowledgment. As discussed in Sec¬ 
tion 4.1, OpenFlow currently does not support an ex¬ 
plicit acknowledgment (ACK) mechanism. In the ab¬ 
sence of ACKs, update procedures are planned accord¬ 
ing to a worst-case analysis (Section 5), both in the 
timed and in the untimed approaches. However, if switches 
are able to notify the controller upon completion of 
an update (as assumed in [18]), then update proce¬ 
dures can sometimes be completed earlier than with¬ 
out using ACKs. Furthermore, ACKs enable updates 
to be performed dynamically [18], whereby at the end 
of each phase the controller dynamically plans the next 
phase. Fortunately, the timed and untimed approaches 
can be combined. For example, in the presence of an 
acknowledgment mechanism, update procedures can be 
performed in a dynamic, untimed, ACK-based manner, 
with a timed garbage collection phase at the end. This 
flexible mix-and-match approach allows the SDN pro¬ 
grammer to enjoy the best of both worlds. 

9. RELATED WORK 

The use of time in distributed applications has been 
widely analyzed, both in theory and in practice. Anal¬ 
ysis of the usage of time and synchronized clocks, e.g., 
Lamport [21, 22] dates back to the late 1970s and early 
1980s. Accurate time has been used in various different 
applications, such as distributed database [10], indus¬ 
trial automation systems [14], automotive networks [15], 
and accurate instrumentation and measurements [36]. 
While the usage of accurate time in distributed systems 
has been widely discussed in the literature, we are not 
aware of similar analyses of the usage of accurate time 
as a means for performing consistent updates in com¬ 
puter networks. 

Time-of-day routing [8] routes traffic to different des¬ 
tinations based on the time-of-day. Path calendaring [19] 
can be used to configure network paths based on sched¬ 
uled or foreseen traffic changes. The two latter examples 


are typically performed at a low rate and do not place 
demanding requirements on accuracy. 

In [12] the authors briefly mentioned that it would be 
interesting to explore using time synchronization to in¬ 
struct routers or switches to change from one configura¬ 
tion to another at a specific time, but did not pursue the 
idea beyond this observation. Our previous work [29, 
30] introduced the concept of using time to coordinate 
updates in SDN. Based on our work [33], the OpenFlow 
protocol [41, 40] currently supports time-based network 
updates. In [34] we presented a practical method to 
implement accurately scheduled network updates. In 
this paper we analyze the use of time in consistent up¬ 
dates, and show that time can improve the scalability 
of consistent updates. 

Various consistent network update approaches have 
been analyzed in the literature. Two of the most well- 
known update methods are the ordered approach [11, 
45, 24, 18], and the two-phase approach [42, 20]. None 
of these works proposed to use accurate time and syn¬ 
chronized clocks as a means to coordinate the updates. 
In this paper we show that time can be used to improve 
these two methods, allowing to reduce the overhead dur¬ 
ing update procedures. 

The analysis of [20] proposed an incremental method 
that improves the scalability of consistent updates by 
breaking each update into multiple independent rounds, 
thereby reducing the total overhead consumed in each 
separate round. The timed approach we present in this 
paper can improve the incremental method even fur¬ 
ther, by reducing the overhead consumed in each round. 

10. CONCLUSION 

Accurate time synchronization has become a common 
feature in commodity switches and routers. We have 
shown that it can be used to implement consistent up¬ 
dates in a way that reduces the update duration and 
the expensive overhead of maintaining duplicate con¬ 
figurations. Moreover, we have shown that accurate 
time can be used to tune the fine tradeoff between con¬ 
sistency and scalability during network updates. Our 
experimental evaluation demonstrates that timed up¬ 
dates allow scalability that would not be possible with 
conventional update methods. 
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A. APPENDIX: DATASET DETAILS 

The measurement results presented in Section 4.1 are 
based on publicly available datasets from [6, 2]. The 
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data we analyzed consists of RTT measurements be¬ 
tween 20 source-destination pairs, listed in Table 4. The 
data is based on measurements taken from November 2013 
to November 2014. 


Source site 

Destination site 

Trace 

source 

ping.desy.de 

ba.sanet.sk 

[6] 

pinger.stanford.edu 

ihep.ac.cn 

[6] 

pinger.stanford.edu 

institutokilpatrick 

.edu 

[6] 

pinger.uet.edu.pk 

ping.cern.ch 

[6] 

pinger2 .if. ufrj. br 

ping.cern.ch 

[6] 

pinger.arn.dz 

dns.sinica.edu.tw 

[6] 

pinger.stanford.edu 

ping.cern.ch 

[6] 

pinger.stanford.edu 

mail.gnet.tn 

[6] 

pinger.stanford.edu 

tg.refer.org 

[6] 

pinger.stanford.edu 

www.unitec.edu 

[6] 

ampz-catalyst 

ampz-citylink 

[2] 

ampz-inspire 

ampz-massey-pn 

[2] 

ampz-netspace 

ampz-inspire 

[2] 

ampz-ns3a 

ampz-citylink 

[2] 

ampz-ns3a 

WWW. stuff. CO. nz 

[2] 

ampz-rurallink 

www.facebook.com 

[2] 

ampz-rurallink 

www.google.co.nz 

[2] 

ampz-waikato 

www.facebook.com 

[2] 

ampz-waikato 

www.google.co.nz 

[2] 

ampz-wxc-akl 

ampz-csotago 

[2] 


Table 4: List of delay measurement traces. 
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